BigDFT.URL module

The URL handler exists to assist connection to a remote system.

As it also wraps file transfer and command execution systems, even non-remote runs should rely on the use of a URL class. For developers: This allows RemoteRunners to focus on the file and job handling aspects

The following examples should give you an idea of basic usage of the URL module. Lets assume we have a remote server we want to run on, with IP address 192.168.0.0, and we have access via user user

URL can be initialised empty then updated with connection details, or produced in a number of ways:

>>> u = URL()
>>> print(u.local)
>>> True
>>> u.user = 'user'
>>> u.host = '192.168.0.0'
>>> print(u.local)
>>> False

Initialised with args from the start:

>>> u = URL(user='user', host='192.168.0.0')
>>> print(u.local)
>>> False

Or built from a “legacy” style string:

>>> u = URL.from_string('user@192.168.0.0')
>>> print(u.local)
>>> False

A URL instance can also connect to an existing SSH tunnel, simply specify your local endpoint. (URL uses the presence of a port to detect tunnels).

>>> u = URL(user='user', host='127.0.0.10', port=1000)
>>> print(u.local)
>>> False
>>> print(u.is_tunnel)
>>> True

For support creating a tunnel connection, URL has the ability to assist by providing the commands needed. Though you must provide the address of the tunnel target for this

>>> u = URL.from_tunnel(user='user', host='192.168.0.0')

You will then be given two commands to enter on the remote:

  • command 1 sets up the tunnel

  • command 2 tests the tunnel

Once this is done, confirm to the tunnel constructor that the tunnel is active with “y” and URL will connect.

See URL.from_tunnel() for more details.

There is also the functionality to test the connection with the remote:

URL.test_connection()

This function, when called, will attempt to create folders on both local and remote systems, then test file transfer functionality between them.

Further details of attributes and methods are below:

class URL(user: Optional[str] = None, host: Optional[str] = None, port: Optional[int] = None, rsync: Optional[str] = None, verbose: bool = False)[source]

Container to store the connection info for a Remote run

Parameters
  • user (str) – username for the remote system

  • host (str) – host address of the remote system

  • port (int, str) – port to connect to for ssh tunnels

classmethod from_string(string: str, rsync: Optional[str] = None, verbose: bool = False)[source]

Create a connection from a user@hostname string

Parameters
  • string (str) – a user@hostname string to create a connection from

  • rsync (str, optional) – initial flags for rsync

  • verbose (bool, optional) – sets verbosity for connection actions

Returns

URL

classmethod from_tunnel(host: str, user: Optional[str] = None, port: Optional[int] = None, endpoint: Optional[str] = None, rsync: Optional[str] = None, background: bool = False)[source]

Create a URL instance based on an ssh tunnel, giving the needed command

Guides the user through tunnel creation, creating the command corresponding to the desired connection details.

Note

It is advised to research and set up the tunnel yourself for your environment, as this method does not adapt per use case. Therefore the provided commands may set up an incorrect (or nonfunctional) tunnel.

Parameters
  • user (str) – username for remote host

  • host (str) – hostname to connect to

  • port (int, optional) – local port to connect to. Defaults to 1800

  • endpoint (str, optional) – local tunnel endpoint. Defaults to 127.0.0.10

  • rsync (str, optional) – rsync flags to be given to URL, if needed

  • background (bool, optional) –

    place the process in the background

    Warning

    The process should be killed when not needed if placed in the background

Returns

URL

When creating a tunnel connection you must specify the address of the remote server.

For example, to connect to our 192.168.0.0 server, the connection details for a standard ssh are required variables. You can optionally specify the tunnel endpoint: The “local” IP address and port of the tunnel.

When creating a tunnel, you will be given two commands:

  • tunnel creation command

  • tunnel test command

The tunnel creation command will look similar to this:

ssh user@192.168.0.0 -L 1000:127.0.0.10:192.168.0.0:22 -N

Whereas the test command will be a simple ssh “ls” command to ensure that the tunnel is operational and allows info transfer.

ssh -p 1000 127.0.0.10 "ls"

Assuming everything works, enter y to continue.

userhost(sep='@')[source]

Return a userhost string, allowing for user to be empty

property user

Attribute for the ‘user’ connection parameter

property host

Attribute for the ‘host’ connection parameter

property port

Attribute for the ‘port’ connection parameter

Used to detect presence of an SSH tunnel

property verbose

Attribute determining the verbosity of operations

property local

Returns True if this connection is local only

property is_tunnel

Returns True if this connection uses an SSH tunnel

property rsync_flags

Attribute determining the flags used for any rsync process

Note

These flags will be ignored if rsync is not available (cp is used as a fallback in this case)

property permissions

Displays the available permissions for this URL

Dictates whether files can be created in root, or only in the user directories

Note

This property will test the connection if no permissions can be found

Returns

permissions

Return type

dict

property ssh_override

returns True if we are overriding the ssh value

cmd(cmd: str, asynchronous: bool = False, local: Optional[bool] = None, verbose: Optional[bool] = None, suppress_warn: bool = False, return_error: bool = False) str[source]

Execute a command

Parameters
  • cmd (str) – the command to be executed

  • asynchronous (bool, optional) – do not wait for stdout if async

  • local (bool, optional) – force local or remote execution

  • verbose (bool, optional) – verbosity override for this execution

  • suppress_warn (bool, optional) – suppress warnings

  • return_error (bool, optional) – prefer returning a raised error over stdout

Returns

stdout if not async, None otherwise

Return type

str

ls(target: str = './', local: bool = False) list[source]

Perform ls command on target

Parameters
  • target (str) – target entity (folder or file)

  • local (bool) – force local or remote search

Returns

newline split result

Return type

list

rsync(files: list, origin: str, target: Optional[str] = None, push: bool = True, dry_run: bool = False)[source]

Create rsync “link” between origin and target, pull or push files

Parameters
  • files (list) – list of filenames to transfer

  • origin (str) – origin folder

  • target (str) – target folder

  • push (bool) – files are to be sent (or received)

  • dry_run (bool) – don’t send the command if True

Returns

the rsync command stdout or the command if dry_run

Return type

str

ensure_dir(dir: str, force_local: bool = False)[source]

Ensure a directory exists by attempting to create it

Parameters
  • dir (str) – directory to create

  • force_local (bool) – force a local run

Warning: Duplicate of futile.Utils.ensure_dir

class URLUtils(parent: BigDFT.URL.URL)[source]

Extra functions to go with the URL class, called via URL.utils

As it requires a parent URL to function, and is instantiated with a URL, there is little to no purpose to using this class exclusively

Parameters

parent (URL) – parent class to provide utils to

search_folder(files: list, folder: str, local: bool = False)[source]

Search folder for files, returning a boolean presence dict

Parameters
  • files (list) – list of filenames to check for. Optionally, a string for a single file

  • folder (str) – folder to scan

  • local (bool) – perform the scan locally (or remotely)

Returns (dict):

{file: present} dictionary

touch(file: str, local: bool = False)[source]

perform unix touch, creating or updating file

Parameters
  • file (str) – filename or path to file

  • local (bool) – force local (or remote) execution

mkdir(file, local=False)[source]

perform unix mkdir -p, creating a folder structure

Parameters
  • file (str) – name or path to folder

  • local (bool) – force local (or remote) execution

ls(file: str, as_list: bool = True, local: bool = False)[source]

Identify the files present on the directory

Parameters
  • file (str) – name or path to folder.

  • as_list (bool) – convert to a list format

  • local (bool) – force local (or remote) execution

class Flags(initial_flags: str = '', prefix: Optional[str] = None)[source]

Basic but flexible handler for terminal command flags

Allows for inplace modification:

>>> f = Flags('abcdd')
>>> f -= 'd'
>>> f.flags
>>> '-abcd'
Parameters
  • initial_flags (str) – initial base flags to be used and modified if needed

  • prefix (optional, str) – the non-argument prefix for these flags (-, , etc.)

property flags

Returns the fully qualified flags as a string

property prefix

Attribute determining the - prefix used for these flags

For example, a flag with a prefix of - will be added as -flag Change the prefix to to be passed as –flag, and so on